All the Models!

Now that we know how to model urges using heart rate, we can compute models for each participant and for the global dataset.

Interesting Queries

What were the best overall predictions?

Here, we order the values prioritising accuracy, followed by sensitivity (ability to correctly detect an urge/episode), and lastly specificity (ability to correctly reject non-urges).

From this, I wonder how we can decide the best model…

What were the best predictions for each participant?

We can see that random forest is quite popular (even though its global prediction was low, as compared to this subset):

According to this plot, the random forest consistently performs best (judging by the mean). It is only outperformed in sensitivity, by the neural net (which has poorest mean accuracy and mean specificity) and the Naive Bayesian Tree (which also has poor mean specificity by comparison).

So what did random forest do?

Random forest was able to achieve 100% prediction results on participant 200, and performed poorly with participant 205.

What makes participants 200 and 205 outliers in random forest’s opinion?

First, we compare the model specifications:

By these plots, we can agree that the variables sdnn, sdann, and sdnnix are probably most important as they are found highest both in the global and best model. Let’s confirm by ranking the variable importance in each model:

[1] "Ranks for variable `hr`:"

----------
 Rank   n 
------ ---
  7     4 

  5     3 

  3     2 

  6     2 

  8     2 

  9     2 

  1     1 

  2     1 

  4     1 
----------

[1] "Ranks for variable `meanrr`:"

----------
 Rank   n 
------ ---
  2     4 

  5     4 

  6     3 

  4     2 

  7     2 

  8     2 

  1     1 
----------

[1] "Ranks for variable `sdnn`:"

----------
 Rank   n 
------ ---
  2     8 

  3     4 

  6     2 

  1     1 

  4     1 

  8     1 

  10    1 
----------

[1] "Ranks for variable `sdann`:"

-----------
 Rank   n  
------ ----
  1     12 

  2     2  

  3     2  

  4     2  
-----------

[1] "Ranks for variable `sdnnix`:"

----------
 Rank   n 
------ ---
  5     4 

  1     3 

  3     3 

  4     2 

  9     2 

  2     1 

  6     1 

  7     1 

  8     1 
----------

[1] "Ranks for variable `pnn50`:"

----------
 Rank   n 
------ ---
  4     4 

  6     4 

  3     3 

  5     3 

  2     1 

  7     1 

  9     1 

  10    1 
----------

[1] "Ranks for variable `sdsd`:"

----------
 Rank   n 
------ ---
  9     5 

  8     4 

  5     2 

  6     2 

  7     2 

  3     1 

  4     1 

  10    1 
----------

[1] "Ranks for variable `rmssd`:"

----------
 Rank   n 
------ ---
  8     6 

  9     4 

  3     2 

  4     2 

  7     2 

  6     1 

  10    1 
----------

[1] "Ranks for variable `irrr`:"

----------
 Rank   n 
------ ---
  7     5 

  9     4 

  4     3 

  6     2 

  8     2 

  5     1 

  10    1 
----------

[1] "Ranks for variable `madrr`:"

-----------
 Rank   n  
------ ----
  10    13 

  2     1  

  3     1  

  5     1  

  6     1  

  7     1  
-----------

We can assume that participant 205 had the poorest predictions because their model selected poor variables. What do these variables look like?

Here’s what the global data set looks like:

[1] 2199   11
       hr             meanrr            sdnn            sdann            sdnnix            pnn50       
 Min.   : 45.13   Min.   : 372.4   Min.   : 24.29   Min.   :  0.00   Min.   :  2.293   Min.   :0.0000  
 1st Qu.: 72.54   1st Qu.: 696.0   1st Qu.: 53.57   1st Qu.: 15.22   1st Qu.: 46.073   1st Qu.:0.1088  
 Median : 79.33   Median : 762.8   Median : 64.39   Median : 26.76   Median : 56.843   Median :0.1606  
 Mean   : 81.03   Mean   : 766.6   Mean   : 69.86   Mean   : 34.90   Mean   : 59.880   Mean   :0.1673  
 3rd Qu.: 87.04   3rd Qu.: 833.5   3rd Qu.: 80.17   3rd Qu.: 45.98   3rd Qu.: 69.031   3rd Qu.:0.2258  
 Max.   :162.51   Max.   :1332.5   Max.   :452.62   Max.   :263.32   Max.   :586.990   Max.   :0.6000  
      sdsd            rmssd             irrr            madrr          Episodes
 Min.   : 18.71   Min.   : 18.70   Min.   : 11.72   Min.   :-39.0647   0:1037  
 1st Qu.: 53.95   1st Qu.: 53.86   1st Qu.: 62.50   1st Qu.:  0.0000   1:1162  
 Median : 66.51   Median : 66.21   Median : 62.50   Median :  0.0000           
 Mean   : 71.63   Mean   : 71.23   Mean   : 77.04   Mean   :  1.0660           
 3rd Qu.: 82.94   3rd Qu.: 82.54   3rd Qu.: 93.75   3rd Qu.:  0.0006           
 Max.   :502.60   Max.   :491.55   Max.   :312.51   Max.   : 93.7538           

Here’s participant 200:

[1] 21 11
       hr             meanrr           sdnn            sdann             sdnnix           pnn50        
 Min.   : 65.34   Min.   :608.8   Min.   : 47.01   Min.   :  2.141   Min.   : 40.13   Min.   :0.07312  
 1st Qu.: 79.25   1st Qu.:685.4   1st Qu.: 60.43   1st Qu.: 20.557   1st Qu.: 56.24   1st Qu.:0.16667  
 Median : 81.15   Median :750.9   Median : 80.69   Median : 32.198   Median : 63.72   Median :0.20183  
 Mean   : 83.08   Mean   :745.1   Mean   : 86.25   Mean   : 49.430   Mean   : 75.14   Mean   :0.20711  
 3rd Qu.: 89.23   3rd Qu.:760.6   3rd Qu.: 87.92   3rd Qu.: 45.186   3rd Qu.: 77.41   3rd Qu.:0.27193  
 Max.   :100.53   Max.   :939.8   Max.   :209.57   Max.   :185.514   Max.   :183.63   Max.   :0.33333  
      sdsd            rmssd             irrr            madrr            Episodes
 Min.   : 42.43   Min.   : 42.42   Min.   : 62.50   Min.   :-23.438573   0: 8    
 1st Qu.: 63.03   1st Qu.: 62.96   1st Qu.: 74.22   1st Qu.:  0.000000   1:13    
 Median : 78.50   Median : 78.31   Median : 93.75   Median :  0.000000           
 Mean   : 80.12   Mean   : 79.78   Mean   : 95.61   Mean   :  1.116191           
 3rd Qu.: 94.27   3rd Qu.: 93.65   3rd Qu.:125.01   3rd Qu.:  0.000954           
 Max.   :144.77   Max.   :143.62   Max.   :156.26   Max.   : 15.625834           

And participant 205:

[1] 15 11
       hr             meanrr           sdnn            sdann            sdnnix          pnn50        
 Min.   : 75.37   Min.   :427.6   Min.   : 53.46   Min.   : 8.355   Min.   :33.23   Min.   :0.09091  
 1st Qu.: 83.17   1st Qu.:663.9   1st Qu.: 76.75   1st Qu.:28.537   1st Qu.:64.08   1st Qu.:0.20000  
 Median : 85.67   Median :707.9   Median : 81.81   Median :37.346   Median :69.13   Median :0.24194  
 Mean   : 92.91   Mean   :677.3   Mean   : 86.12   Mean   :45.911   Mean   :69.45   Mean   :0.22682  
 3rd Qu.: 91.22   3rd Qu.:733.3   3rd Qu.: 95.51   3rd Qu.:49.529   3rd Qu.:78.48   3rd Qu.:0.26491  
 Max.   :142.86   Max.   :813.2   Max.   :118.79   Max.   :99.376   Max.   :96.78   Max.   :0.28671  
      sdsd            rmssd             irrr            madrr            Episodes
 Min.   : 57.15   Min.   : 54.58   Min.   : 39.06   Min.   :-31.250715   0:7     
 1st Qu.: 69.55   1st Qu.: 69.29   1st Qu.: 78.13   1st Qu.: -7.812500   1:8     
 Median : 84.69   Median : 84.01   Median : 93.75   Median :  0.000238           
 Mean   : 85.09   Mean   : 84.38   Mean   : 93.75   Mean   : -2.083309           
 3rd Qu.: 91.83   3rd Qu.: 91.28   3rd Qu.:115.24   3rd Qu.:  3.907025           
 Max.   :133.22   Max.   :131.84   Max.   :156.26   Max.   : 15.625000           

We can examine the distributions of the three top importance variables too:

Notice the difference in the ranges of these variables.

In any case, both participants 200 and 205 have extremely small data sets; changing any parameters such as the random seed could easily invalidate these results. Let’s examine the second best predictions, participant 207 (who we examined before).

Participant 207

Once again, our top three variables are present.

I’m noticing that the good models all lack heavy tails. Participant 207’s data is very similar to 200, however, it has a tiny bump way out at 400 for sdnn; by comparison, participant 200 had a range within 250 for these three variables. Does this mean that our preprocessing should apply a ceiling to these three variables?

How do these important variables change for each event?

How are the decision trees structured?

Random forest is a simple algorithm with a complex explanation; it’s considered one of the more black box algorithms, as it’s not common to get too much detail about mechanics out of it. Nevertheless, we can derive some information by randomly selecting a tree and checking it’s mechanics. This is not how the overall model works though (the real algorithm draws samples repeatedly and builds on the errors of small decision trees).